{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c0a95872",
   "metadata": {},
   "source": [
    "# HyDE\n",
    "For a given query, HyDE retrieval pipeline contains 4 components:\n",
    "1. Promptor: bulid prompt for generator based on specific task.\n",
    "2. Generator: generates hypothesis documents using Large Language Model.\n",
    "3. Encoder: encode hypothesis documents to HyDE vector.\n",
    "4. Searcher: search nearest neighbour for the HyDE vector (dense retrieval)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1ee489b",
   "metadata": {},
   "source": [
    "### Initialize HyDE components\n",
    "We use [pyserini](https://github.com/castorini/pyserini) as the search interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "65c24913",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from pyserini.search import FaissSearcher, LuceneSearcher\n",
    "from pyserini.search.faiss import AutoQueryEncoder\n",
    "\n",
    "from hyde import Promptor, OpenAIGenerator, CohereGenerator, HyDE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "dcb0db43",
   "metadata": {},
   "outputs": [],
   "source": [
    "KEY = '<api key>' # replace with your API key, it can be OpenAI api key or Cohere api key\n",
    "promptor = Promptor('web search')\n",
    "generator = OpenAIGenerator('text-davinci-003', KEY)\n",
    "encoder = AutoQueryEncoder(encoder_dir='facebook/contriever', pooling='mean')\n",
    "searcher = FaissSearcher('contriever_msmarco_index/', encoder)\n",
    "corpus = LuceneSearcher.from_prebuilt_index('msmarco-v1-passage')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0fe1eb3",
   "metadata": {},
   "source": [
    "### Build a HyDE pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "ccacadc5",
   "metadata": {},
   "outputs": [],
   "source": [
    "hyde = HyDE(promptor, generator, encoder, searcher)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ee99483",
   "metadata": {},
   "source": [
    "### Load example Query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "b8d1a85c",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = 'how long does it take to remove wisdom tooth'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c3be967",
   "metadata": {},
   "source": [
    "### Build Zeroshot Prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "a6e0eecc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Please write a passage to answer the question.\n",
      "Question: how long does it take to remove wisdom tooth\n",
      "Passage:\n"
     ]
    }
   ],
   "source": [
    "prompt = hyde.prompt(query)\n",
    "print(prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1821d8b9",
   "metadata": {},
   "source": [
    "### Generate Hypothesis Documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "4032c2b7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HyDE Generated Document: 0\n",
      "There is no one-size-fits-all answer to this question, as the time it takes to remove a wisdom tooth can vary depending on the individual case. In general, however, the procedure usually takes around 30 minutes to an hour to complete.\n",
      "HyDE Generated Document: 1\n",
      "It generally takes around 30 to 45 minutes to remove a wisdom tooth. However, the time may vary depending on the position of the tooth and the amount of work required.\n",
      "HyDE Generated Document: 2\n",
      "It usually takes around 30 to 45 minutes to remove a wisdom tooth. However, the length of time may vary depending on the individual case.\n",
      "HyDE Generated Document: 3\n",
      "It can take anywhere from a few days to a few weeks to remove a wisdom tooth. The length of time will depend on the individual case and the severity of the tooth.\n",
      "HyDE Generated Document: 4\n",
      "The length of time it takes to remove a wisdom tooth varies depending on the tooth's position and the amount of bone surrounding it. The procedure can take anywhere from 20 minutes to an hour.\n",
      "HyDE Generated Document: 5\n",
      "The length of time it takes to remove a wisdom tooth depends on a few factors, such as the position of the tooth and the type of extraction (simple or surgical). A simple extraction is typically quicker, taking about 20 minutes, while a surgical extraction can take up to an hour.\n",
      "HyDE Generated Document: 6\n",
      "It can take anywhere from a few days to a couple of weeks to recover from having your wisdom teeth removed. The actual procedure itself is usually over within an hour or so, but it can take some time for the numbing medication to wear off and for the swelling to go down.\n",
      "HyDE Generated Document: 7\n",
      "It generally takes around 20 minutes to remove a wisdom tooth. However, the time may vary depending on the individual case. In some cases, the tooth may be removed in just a few minutes, while in others it may take up to an hour.\n"
     ]
    }
   ],
   "source": [
    "hypothesis_documents = hyde.generate(query)\n",
    "for i, doc in enumerate(hypothesis_documents):\n",
    "    print(f'HyDE Generated Document: {i}')\n",
    "    print(doc.strip())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "022a4cd8",
   "metadata": {},
   "source": [
    "### Encode HyDE vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "cdb4e5b9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 768)\n"
     ]
    }
   ],
   "source": [
    "hyde_vector = hyde.encode(query, hypothesis_documents)\n",
    "print(hyde_vector.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe95165a",
   "metadata": {},
   "source": [
    "### Search Relevant Documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "8ae97c0a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HyDE Retrieved Document: 0\n",
      "4174313\n",
      "The time it takes to remove the tooth will vary. Some procedures only take a few minutes, whereas others can take 20 minutes or longer. After your wisdom teeth have been removed, you may experience swelling and discomfort, both on the inside and outside of your mouth.This is usually worse for the first three days, but it can last for up to two weeks. Read more about how a wisdom tooth is removed and recovering from wisdom tooth removal.he time it takes to remove the tooth will vary. Some procedures only take a few minutes, whereas others can take 20 minutes or longer. After your wisdom teeth have been removed, you may experience swelling and discomfort, both on the inside and outside of your mouth.\n",
      "HyDE Retrieved Document: 1\n",
      "18103\n",
      "Before having your wisdom teeth removed, you'll be given an injection of local anaesthetic to numb the tooth and surrounding area. If you're particularly anxious about the procedure, your dentist or surgeon may give you a sedative to help you relax. This usually involves an injection into your arm.urgery to remove wisdom teeth shouldn't be painful, because the area will be numb. However, if you feel pain during the procedure, tell your dentist or oral surgeon so they can give you more anaesthetic. How long it takes to remove the tooth will vary.\n",
      "HyDE Retrieved Document: 2\n",
      "91493\n",
      "The time it takes to remove the tooth will vary. Some procedures only take a few minutes, whereas others can take 20 minutes or longer. After your wisdom teeth have been removed, you may experience swelling and discomfort, both on the inside and outside of your mouth. This is usually worse for the first three days, but it can last for up to two weeks. Read more about how a wisdom tooth is removed and recovering from wisdom tooth removal.\n",
      "HyDE Retrieved Document: 3\n",
      "4155912\n",
      "How long does it take to remove all wisdom teeth?   I got my wisdom teeth removed 5 days ago. I received intravenous anesthesia, so I was not concious during the process, but those present said it only took about 35 to 40â¦ minutes for removal.\n",
      "HyDE Retrieved Document: 4\n",
      "7344529\n",
      "Complications like infection can lengthen the time it takes to heal up, but here is a general timeline: 1  Swelling and pain will be the greatest during the first 3 days (peaking at about 48hours). 2  Normally, the sockets should take about 2 weeks to 1 month to cover over with solid gum tissue after scabbing first.\n",
      "HyDE Retrieved Document: 5\n",
      "4174308\n",
      "How wisdom teeth are removed. Your dentist may remove your wisdom teeth or they may refer you to a specialist surgeon for hospital treatment. Before the procedure, you'll usually be given a local anaesthetic injection to numb the area around the tooth.he time it takes to remove the tooth will vary. Some procedures only take a few minutes, whereas others can take 20 minutes or longer. After your wisdom teeth have been removed, you may experience swelling and discomfort, both on the inside and outside of your mouth.\n",
      "HyDE Retrieved Document: 6\n",
      "4174310\n",
      "If your dentist thinks you may need your wisdom teeth removed, they'll usually carry out an X-ray of your mouth. This gives them a clearer view of the position of your teeth. As with any teeth problems, it's important to see your dentist as soon as possible, rather than waiting for your regular dental check-up.he time it takes to remove the tooth will vary. Some procedures only take a few minutes, whereas others can take 20 minutes or longer. After your wisdom teeth have been removed, you may experience swelling and discomfort, both on the inside and outside of your mouth.\n",
      "HyDE Retrieved Document: 7\n",
      "4595794\n",
      "If you have a dentist remove your wisdom teeth, i can take between 20 minutes and three hours per tooth. If you get a surgeon to remove your wisdom teeth, it takes between 30 seconds and five minutes per tooth. To find out more on wisdom teeth surgery and the approximate times on surgery as well as post-operative instructions, go to http://dentalimplantsaustralia.com/. Source(s): http://dentalimplantsaustralia.com/.\n",
      "HyDE Retrieved Document: 8\n",
      "4159345\n",
      "How long does it take to heal after your wisdom teeth are removed. As with any type of surgery, everyone heals differently, but it usually takes between 10 days to 2 weeks for wisdom teeth removal.\n",
      "HyDE Retrieved Document: 9\n",
      "2940180\n",
      "You shouldn't even brush your teeth for the first day of recovery. According to the offices of practicing oral surgeon Dr. Joseph Arzadon of Arlington, Virginia, typical wisdom teeth recovery time is three to four days, although it can be as long as one week. The length of recovery depends a lot on how badly the wisdom teeth were impacted and how they were erupting.\n"
     ]
    }
   ],
   "source": [
    "hits = hyde.search(hyde_vector, k=10)\n",
    "for i, hit in enumerate(hits):\n",
    "    print(f'HyDE Retrieved Document: {i}')\n",
    "    print(hit.docid)\n",
    "    print(json.loads(corpus.doc(hit.docid).raw())['contents'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2edb68c",
   "metadata": {},
   "source": [
    "### End to End Search\n",
    "\n",
    "e2e search will directly go through all the steps descripted above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "47e9f353",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HyDE Retrieved Document: 0\n",
      "4174313\n",
      "The time it takes to remove the tooth will vary. Some procedures only take a few minutes, whereas others can take 20 minutes or longer. After your wisdom teeth have been removed, you may experience swelling and discomfort, both on the inside and outside of your mouth.This is usually worse for the first three days, but it can last for up to two weeks. Read more about how a wisdom tooth is removed and recovering from wisdom tooth removal.he time it takes to remove the tooth will vary. Some procedures only take a few minutes, whereas others can take 20 minutes or longer. After your wisdom teeth have been removed, you may experience swelling and discomfort, both on the inside and outside of your mouth.\n",
      "HyDE Retrieved Document: 1\n",
      "91493\n",
      "The time it takes to remove the tooth will vary. Some procedures only take a few minutes, whereas others can take 20 minutes or longer. After your wisdom teeth have been removed, you may experience swelling and discomfort, both on the inside and outside of your mouth. This is usually worse for the first three days, but it can last for up to two weeks. Read more about how a wisdom tooth is removed and recovering from wisdom tooth removal.\n",
      "HyDE Retrieved Document: 2\n",
      "4155912\n",
      "How long does it take to remove all wisdom teeth?   I got my wisdom teeth removed 5 days ago. I received intravenous anesthesia, so I was not concious during the process, but those present said it only took about 35 to 40â¦ minutes for removal.\n",
      "HyDE Retrieved Document: 3\n",
      "18103\n",
      "Before having your wisdom teeth removed, you'll be given an injection of local anaesthetic to numb the tooth and surrounding area. If you're particularly anxious about the procedure, your dentist or surgeon may give you a sedative to help you relax. This usually involves an injection into your arm.urgery to remove wisdom teeth shouldn't be painful, because the area will be numb. However, if you feel pain during the procedure, tell your dentist or oral surgeon so they can give you more anaesthetic. How long it takes to remove the tooth will vary.\n",
      "HyDE Retrieved Document: 4\n",
      "7344529\n",
      "Complications like infection can lengthen the time it takes to heal up, but here is a general timeline: 1  Swelling and pain will be the greatest during the first 3 days (peaking at about 48hours). 2  Normally, the sockets should take about 2 weeks to 1 month to cover over with solid gum tissue after scabbing first.\n",
      "HyDE Retrieved Document: 5\n",
      "3654735\n",
      "All surgery is associated with some degree of pain, from mild to severe. If your surgery was more extensive, then you will have more post-operative pain, unfortunately. Typically the pain intensity peaks 6 to 10 hours after the procedures.\n",
      "HyDE Retrieved Document: 6\n",
      "4174308\n",
      "How wisdom teeth are removed. Your dentist may remove your wisdom teeth or they may refer you to a specialist surgeon for hospital treatment. Before the procedure, you'll usually be given a local anaesthetic injection to numb the area around the tooth.he time it takes to remove the tooth will vary. Some procedures only take a few minutes, whereas others can take 20 minutes or longer. After your wisdom teeth have been removed, you may experience swelling and discomfort, both on the inside and outside of your mouth.\n",
      "HyDE Retrieved Document: 7\n",
      "4159345\n",
      "How long does it take to heal after your wisdom teeth are removed. As with any type of surgery, everyone heals differently, but it usually takes between 10 days to 2 weeks for wisdom teeth removal.\n",
      "HyDE Retrieved Document: 8\n",
      "2940180\n",
      "You shouldn't even brush your teeth for the first day of recovery. According to the offices of practicing oral surgeon Dr. Joseph Arzadon of Arlington, Virginia, typical wisdom teeth recovery time is three to four days, although it can be as long as one week. The length of recovery depends a lot on how badly the wisdom teeth were impacted and how they were erupting.\n",
      "HyDE Retrieved Document: 9\n",
      "4595794\n",
      "If you have a dentist remove your wisdom teeth, i can take between 20 minutes and three hours per tooth. If you get a surgeon to remove your wisdom teeth, it takes between 30 seconds and five minutes per tooth. To find out more on wisdom teeth surgery and the approximate times on surgery as well as post-operative instructions, go to http://dentalimplantsaustralia.com/. Source(s): http://dentalimplantsaustralia.com/.\n"
     ]
    }
   ],
   "source": [
    "hits = hyde.e2e_search(query, k=10)\n",
    "for i, hit in enumerate(hits):\n",
    "    print(f'HyDE Retrieved Document: {i}')\n",
    "    print(hit.docid)\n",
    "    print(json.loads(corpus.doc(hit.docid).raw())['contents'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "578d2ca9",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.13 ('binder')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "69dc8d5aafec8ae3fa1fc52986190fb7b1bcb1c4684e5d3b6aa96016ecada253"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
